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Abstract 

A  method  for  assessing  situation  awareness  (SA)  in  field  command  and  control 
exercises  is  described  in  this  paper.  This  is  an  adaptation  of  the  Situation  Awareness 
Global  Assessment  Technique  (Endsley,  1995a),  taking  into  account  specific 
constraints  for  use  in  a  field  exercise,  such  as  minimising  the  level  of  intrusiveness. 
This  constructible  assessment  for  SA  (CASA)  requires  that  military  subject  matter 
experts  observe  exercise  events,  construct  specific  and  relevant  questions  to  probe  the 
situational  knowledge  of  the  exercise  participants  and  decide  upon  the  suitable 
timings  for  administration  of  these  questions.  CASA  was  tested  in  a  division-level 
army  C2  exercise.  An  objective  of  the  exercise  was  to  investigate  the  issues  of 
distributed  command  teams,  using  situation  awareness  as  one  of  the  measures.  Having 
applied  CASA  in  this  exercise,  the  strengths  and  limitations  of  CASA  were  identified. 
To  analyse  the  findings,  several  methods  of  computing  situation  awareness  of  teams 
were  utilised.  It  was  found  that  CASA  yielded  results  consistent  with  expectations 
stemming  from  exercise  roles  and  events.  Further  validation  and  refinement  of  CASA 
would  improve  its  utility  and  efficacy  as  a  tool  to  measure  objective  SA  in  a  field 
exercise. 


Introduction 

Situation  awareness  (SA)  can  be  defined  in  many  ways.  To  some,  SA  essentially 
comprises  “knowledge  relevant  to  the  task  being  performed...  (and)  is  a  critical 
component  of  decision  making  (Gawron,  2000)”.  It  can  also  be  construed  as  a  process 
that  consists  of  “keeping  track  of  the  prioritised  significant  events  and  conditions  in 
one’s  environment  (Oliver,  1990)”. 

This  paper  describes  a  method  designed  to  objectively  assess  the  SA  of  personnel 
within  the  constraints  of  a  field  exercise.  Other  attempts  to  measure  SA  in  a  field 
environment  have  utilized  subjective  scales  and  observer  ratings  (Matthews  &  Beal, 
2002).  However,  the  research  team  chose  to  adapt  SAGAT,  an  objective  measure  of 
SA  (Endsley,  1995a)  because  of  its  distinct  advantages  over  subjective  measures.  This 
adapted  form  of  SAGAT  is  named  CASA  (Constructible  Assessment  for  Situation 
Awareness)  for  ease  of  reference  in  this  paper. 

CASA  is  based  on  Endsley ’s  model  of  SA  (1995b)  which  separates  SA  into  three 
levels,  namely:  Level  1,  perception  of  elements  in  the  environment  within  a  volume  of 


time  and  space;  Level  2,  the  comprehension  of  their  meaning  and  integration  of 
multiple  pieces  of  information;  Level  3,  the  projection  of  the  elements’  future  status 
and  ability  to  forecast  future  situation  dynamics.  It  consists  of  a  set  of  questions  that 
objectively  probe  situational  knowledge  of  the  subjects  at  selected  points  in  time.  The 
questions  are  constructed  from  a  list  of  pre-defined  question  templates,  with  the 
option  to  create  new  questions  where  necessary.  Various  measures  of  SA  can  be 
quantified  based  on  subjects’  responses  to  the  questions. 

An  attempt  to  validate  CASA  in  an  army  command  and  control  (C2)  exercise  by  the 
research  team  is  detailed  in  this  paper.  A  previous  paper  describing  a  similar 
undertaking  (French  &  Hutchinson,  2002)  arising  from  similar  motivations  was  found 
during  the  writing  of  this  paper,  after  CASA  had  been  developed  and  tested.  However, 
differences  exist  in  terms  of  methodological  details  and  scope  of  analysis;  these 
differences  will  be  highlighted  where  relevant  in  the  paper. 


Background  Of  Exercise 

The  exercise  was  a  division-level  army  C2  exercise  investigating  the  effects  of  a 
distributed  C2  setup.  Traditionally,  command  teams  are  co-located  in  a  physical 
command  post  from  where  the  battle  is  controlled.  An  obvious  advantage  of  such  an 
arrangement  is  the  ability  to  leverage  upon  physical  proximity  to  facilitate 
information  exchange  and  increase  speed  and  quality  of  communication.  This  in  turn 
enables  members  of  the  team  to  efficiently  and  effectively  achieve  an  understanding 
of  commander’s  intent  and  arrive  at  a  common  operational  picture.  Today,  with 
advanced  concepts  such  as  network-centric  warfare,  command  posts  are  moving 
towards  distributed  models  that  feature  the  added  benefits  of  mobility,  flexibility  and 
agility.  However,  problems  such  as  miscommunication  and  errors  due  to  electronic 
messaging,  as  well  as  a  loss  of  trust  when  individuals  are  not  in  direct  contact  or  view 
of  each  other,  may  arise  in  distributed  environments.  Also,  it  remains  to  be  tested 
whether  the  communication  advantages  of  physical  co-location  are  retained  in  a 
distributed  environment. 

In  the  exercise,  three  command  teams  were  placed  at  different  physical  locations,  the 
independent  variable  being  the  physical  proximity  to  Superior  HQ  which  had  overall 
responsibility  for  the  entire  area  of  operations.  The  area  of  operations  was  partitioned 
into  two  and  each  placed  under  the  command  of  Team  A  and  B:  Team  A  and  HQ 
were  co-located  whereas  Team  B  was  located  in  a  separate  building  about  100m 
away.  The  three  teams  used  the  same  command  and  control  information  systems 
linked  via  network  and  there  was  no  difference  in  their  access  to  communication  tools 
such  as  the  telephone.  No  restriction  was  placed  on  the  movements  of  the  participants. 

CASA  was  used  in  the  exercise  as  part  of  a  larger  scale  cognitive  assessment  to  assess 
the  impact  of  distributed  versus  co-located  command  teams.  In  addition  to  CASA,  the 
assessment  comprised  subjective  workload  and  subjective  SA  measures  and 
video/audio  records  of  communication  activity.  This  paper  reports  the  design  of 
CASA,  its  method  of  administration  in  the  exercise  and  findings;  the  other  measures 
will  be  discussed  in  a  follow-up  paper. 


Constructible  Assessment  For  Situation  Awareness  (CASA) 

CASA  is  adapted  from  the  SAGAT  methodology,  a  well-validated  method  of 
measuring  SA  objectively  (Endsley  &  Garland,  2000)  that  has  been  applied 
extensively  to  military  settings.  However,  the  research  team  felt  that  certain  aspects  of 
the  technique  could  be  modified  for  better  applicability  to  an  army  field  exercise  with 
its  inherent  constraints. 

Firstly,  frequent  administration  of  SAGAT  questions  at  random  times  is  not  always 
possible  because  of  the  level  of  intrusiveness  imposed.  The  exercising  unit  may  not  be 
prepared  to  have  too  many  interruptions  to  the  progression  of  its  exercise  and  the 
blanking  out  of  display  screens  for  a  SAGAT  administration  may  be  deemed  too 
jarring.  Furthermore,  in  a  typical  field  exercise,  participants  would  be  walking  around 
the  exercise  area  and  may  not  be  present  for  a  SAGAT  administration  if 
administration  times  were  randomly  chosen  and  displayed  on  a  terminal. 

Secondly,  the  random  selection  of  SA  questions  from  a  pool  of  prepared  questions 
could  result  in  “wasted”  probes  not  relevant  to  the  unfolding  exercise  events  at  the 
time  of  administration.  While  random  selection  has  its  advantages  of  statistical 
validity  and  objectivity,  but  if  exercise  conditions  do  not  allow  for  a  large  number  of 
SAGAT  administrations,  every  administration  should  be  designed  to  count. 

In  order  to  circumvent  these  constraints,  CASA  was  developed  by  retaining  the 
structure  of  SAGAT  with  several  modifications  to  both  methodology  and  subsequent 
analysis.  As  in  SAGAT,  CASA  involves  the  creation  of  a  list  of  SA  questions  prior  to 
the  exercise,  from  which  a  subset  of  relevant  SA  questions  are  selected  for  each 
administration.  However,  there  are  several  key  differences  from  SAGAT. 

CASA  Features 

CASA  places  emphasis  on  tailoring  each  administration  to  ensure  that  questions 
relevant  to  the  exercise  context  at  that  time  and  its  near  future  will  be  asked,  rather 
than  from  a  random  selection.  Military  Subject  Matter  Experts  (SMEs)  construct 
specific  CASA  questions  by  first  considering  how  events  had  unfolded  and  would 
unfold  in  the  exercise,  then  selecting  the  desired  questions  from  the  list  of  question 
templates.  The  selected  question  templates  would  then  be  “fleshed  out”  with  the 
appropriate  unit  names,  attributes  or  relationships  of  interest.  CASA  is  not  a  closed 
tool;  the  list  of  question  templates  can  be  supplemented  with  new  questions  created 
during  the  exercise  in  response  to  novel  and  unanticipated  situations.  These  new 
questions  can  then  be  generalised  and  added  to  the  list  of  question  templates, 
progressively  making  CASA  more  comprehensive  in  coverage. 

CASA  allows  the  flexibility  in  selection  of  administration  timings  to  be  done  by 
SMEs  instead  of  being  randomised.  However,  caution  needs  to  be  taken  to  ensure  that 
the  administration  times  do  not  follow  an  overt  pattern  (e.g.  during  rest  periods)  so  as 
to  prevent  the  subjects  from  pre-empting  and  preparing  for  an  administration. 

The  selection  flexibility  of  the  question  templates  and  administration  timings  endows 
CASA  with  a  diagnostic  capability,  as  French  and  Hutchinson  (2002)  have  also  noted. 
For  example,  if  the  military  SMEs  observe  or  suspect  that  there  may  be  an 


information  bottleneck  at  a  certain  point  in  time,  a  selection  of  CASA  questions  can 
be  specifically  chosen  to  investigate  if  this  is  true.  Such  a  capability  makes  CASA  a 
more  powerful  tool,  able  to  probe  into  situations  in  addition  to  being  an  objective 
measure  of  SA. 

Another  way  in  which  CASA  is  administered  differently  from  SAGAT  is  the  posing 
of  common  questions  across  different  roles,  unlike  SAGAT  where  subjects  in 
different  roles  are  given  unique  sets  of  questions.  While  it  may  be  argued  that  this 
may  result  in  irrelevant  questions  for  some  subjects,  the  research  team  believes  that  an 
awareness  of  information  outside  one’s  focused  sphere  of  operation  allows  for  better 
appreciation  of  the  overall  situation,  e.g.  the  constraints  and  problems  faced  by  team¬ 
mates.  Also,  since  all  subjects  answered  a  common  set  of  questions  at  every 
administration,  it  is  possible  to  compare  across  subjects  at  each  administration  as  well 
as  at  an  aggregate  level. 

CASA  Questions 

Table  1  shows  examples  of  CASA  question  templates.  There  are  several  question 
types  that  are  used  in  CASA:  multiple-choice  questions,  open-ended  and  map-based. 
A  question  template  is  “fleshed  out”  by  replacing  the  placeholders  in  square  brackets 
with  concrete  terms.  The  correct  answers  to  each  question  should  be  noted  down  by 
referring  to  ground  truth  at  the  appropriate  times. 


Situation 

Awareness 

Level 

Sample  Question  Template 

Answer  Type 

Level  1 

1.  Mark  the  current  location  of  [hostile/friendly  unit]  on  the 
map. 

1.  Marking  on  map. 

2.  What  is  the  current  force  size  of  [hostile/friendly  unit]? 

2.  Open-ended  but 
constrained  by 
context. 

Level  2 

1 .  What  is  the  most  critical  additional  asset  that  [friendly  unit] 
requires  to  carry  out  its  mission? 

2.  Which  one  of  these  hostile  units  currently  poses  the  highest 
threat  priority  to  this  [friendly  unit]  ? 

1 .  Open-ended  but 
constrained  by 
context 

2.  Multiple  Choice 

Level  3 

1.  When  is  the  earliest  projected  time  for  the  securing  of 
[location]? 

1 .  Multiple  Choice 

2.  Is  [hostile  unit]  likely  to  be  in  contact  with  friendly  unit(s) 
by  [time]? 

2.  Yes  /  No 

Table  1  -  Examples  of  CASA  question  templates 


Measures  Of  Situation  Awareness 

The  following  measures  of  SA  can  be  obtained  from  the  results  of  CASA. 

Individual  SA  is  the  percentage  of  correct  responses  an  individual  makes  to  the  SA 
questions  posed.  Group  SA  is  the  level  of  situation  awareness  in  a  team  of 
individuals  as  a  whole,  which  is  the  percentage  of  the  total  number  of  correct 
responses  by  all  members  of  the  team.  Shared  SA  is  an  indication  of  how  two  or 
more  individuals  share  an  awareness  of  the  situation  among  themselves,  computed  by 
the  percentage  of  similarly  answered  questions  by  all  individuals.  Shared  SA  can  be 


based  on  similarly  right  and/or  wrong  responses  depending  on  the  desired  analysis. 
Complementary  SA  is  another  way  of  measuring  the  SA  of  a  team,  where  not  all 
members  are  expected  to  have  common  knowledge  about  the  situation.  It  is  assumed 
that  complementary  roles  within  the  team  cover  each  other  and  information  is 
volunteered  and  shared  freely.  For  Complementary  SA,  as  long  as  one  member  of  the 
team  responds  correctly,  the  question  is  considered  correctly  answered  for  the  entire 
team.  Table  2  illustrates  how  levels  of  individual,  group,  shared  and  complementary 
situation  awareness  are  computed. 


Individual  X  SA  =  3  5  =  60% 

Group  XY  SA  =  7  h-  10  =  70% 

Group  XYZ  SA  =  9  15  =  60% 

Shared*  XY  SA  =  4  h-  5  =  80% 

Shared*  YZ  SA  =  2  5  =  40% 

Shared*  XYZ  SA  =  1  -s-  5  =  20% 
Complementary  SA  of  XYZ  =  4  +  5  =  80% 

*  In  this  illustration,  wrong  answers  contribute  to  Shared  SA  if  respondents  answered  similarly 


Question 

X’s 

answer 

Y’s 

answer 

Z’s 

answer 

1 

(a)  Z 

(a)  Z 

(c)* 

2 

(b)  * 

(b)  * 

(C)* 

3 

(a)  Z 

(a)  Z 

(a)  Z 

4 

(c)* 

(d)Z 

(d)  Z 

5 

(d)Z 

(d)  Z 

(b)  * 

Table  2  -  Computation  of  Levels  of  Situation  Awareness 


CASA  In  A  Distributed  Environment 

CASA  was  first  fielded  as  a  measure  of  situation  awareness  in  the  army  exercise 
described  in  this  paper.  As  the  exercise  was  not  conducted  primarily  for  the  purpose 
of  our  research,  the  administration  was  subject  to  numerous  constraints.  The  exercise 
was  conducted  in  2003  for  duration  of  36  hours  over  3  days. 

Pre-Exercise  Preparation 

A  month  before  the  exercise,  a  list  of  assessment  question  templates  was  created  by 
the  research  team  via  interviews  with  two  military  SMEs.  Regretfully,  a  cognitive  task 
analysis  was  not  performed  because  of  a  lack  of  time.  The  purpose  of  the  interviews 
was  to  identify  information  requirements  of  commanders  in  accomplishing  their  tasks. 
As  per  Endsley’s  three  levels  of  situation  awareness,  the  information  requirements 
were  then  translated  into  templates  for  constructing  questions  pertaining  to  Perception 
(Level  1  SA),  Comprehension  (Level  2  SA)  and  Projection  (Level  3  SA). 

There  were  43  questions  in  the  template,  with  additional  ones  later  created  by  military 
SMEs  during  the  exercise  in  response  to  unanticipated  situations.  This  was  a 
sufficiently  large  question  base  to  avoid  repetition  so  as  to  prevent  subjects  from 
focusing  on  specific  aspects  of  the  situation  during  the  exercise. 

9  subjects  were  identified  for  the  CASA  administration  based  on  their  roles:  Head 
Operations,  Head  Intelligence  and  Commander  of  Teams  A,  B  and  Superior  HQ. 
Another  reason  the  sample  was  kept  small  was  the  effort  required  to  identify  the  SA 
requirements  for  each  role  given  the  short  preparation  time  afforded. 


CASA  Administration 


A  total  of  5  CASA  administrations  took  place  in  the  exercise.  With  the  research  team 
were  two  military  SMEs  observing  the  exercise  events.  They  would  assess  if  the 
developing  situation  was  sufficiently  eventful  to  be  suitable  for  a  CASA 
administration.  If  so,  a  CASA  questionnaire  would  be  constructed  by  the  SMEs 
together  with  the  research  team,  by  identifying  when  in  the  future  to  administer  and 
constructing  SA  questions  accordingly.  This  generally  took  approximately  1-2  hours 
from  the  time  it  was  decided  to  conduct  an  administration  to  the  actual  administration. 
There  were  occasions  when  a  planned  administration  was  aborted  due  to  a  sudden  or 
unanticipated  change  in  scenario  by  Exercise  Control.  The  medium  of  administration 
was  printed  hardcopies  of  the  questionnaire  with  an  attached  exercise  map  for 
answering  questions  regarding  locations,  as  well  as  a  cover  instruction  sheet.  The 
paper  medium  was  chosen  for  the  exercise  since  displays  were  not  to  be  blanked  out 
for  an  administration.  There  was  also  a  lack  of  time  to  develop  other  electronic  means 
of  administration. 

The  CASA  questionnaires  were  handed  out  to  the  9  respondents  within  5  minutes  of 
the  administration  time  that  was  decided  upon.  At  the  same  time,  the  ground  truth  was 
noted  down  for  grading  purposes.  Respondents  were  given  5  minutes  to  complete  the 
questionnaires  to  discourage  over-analyzing  the  questions  and  minimise  the  effects  of 
memory  decay.  Unlike  the  protocol  employed  by  French  and  Hutchinson  (2002),  the 
exercise  was  not  frozen  during  these  administrations.  This  avoided  imposing  too  large 
a  disruption  but  limited  the  sample  size  that  could  be  surveyed  simultaneously.  Upon 
collection  of  the  questionnaires,  grading  was  done  against  the  ground  truth  noted 
down  earlier. 

The  diagnostic  capability  of  CASA  proved  useful  during  the  exercise  when  it  was 
known  that  the  status  of  an  exercise  task  force  may  be  ambiguous  at  a  certain  point  in 
time.  The  SMEs  wanted  to  find  out  which  of  the  teams  had  accurate  knowledge 
regarding  the  task  force.  By  specifying  appropriate  questions,  it  was  discovered  that 
Team  A  and  HQ  were  correctly  informed  whereas  Team  B  (the  distributed  team)  was 
still  operating  under  wrong  assumptions  about  the  task  force. 

An  analysis  of  the  questions  administered  during  the  exercise  (a  total  of  47  questions 
across  the  5  administrations)  showed  that  approximately  7  question  types  were 
particularly  favored  by  the  SMEs  and  used  repeatedly  with  slight  variation.  Reasons 
for  this  could  be  unfamiliarity  on  the  part  of  the  SMEs  with  the  CASA  methodology, 
and  hence  reluctance  to  go  through  all  question  types  in  the  template.  It  may  also  be 
due  to  the  small  number  of  administrations  or  the  exercise  scenario  for  the  other 
questions  to  be  relevant.  Only  a  small  proportion  of  the  questions  (less  than  10%)  was 
novel  i.e.  created  outside  the  pre-defined  types  in  the  template. 


Results 


Responses  were  analysed  in  terms  of  individual,  group,  shared  and  complementary 
SA  as  described  in  Table  2. 

Individual  and  Group  SA  Levels 


Figure  1  -  Group  SA  levels  by  Team  for  all  5  administrations 

Figure  1  presents  the  Group  SA  Levels  of  Superior  HQ,  Team  A  and  Team  B  over  all 
5  administrations.  Group  SA  in  HQ  is  higher  than  that  of  Team  A,  which  in  turn  is 
higher  than  Team  B.  It  is  logical  that  the  SA  of  HQ  would  be  higher  than  either  team 
because  HQ  oversees  the  entire  area  of  operation.  Furthermore,  the  literature  suggests 
that  SA  is  correlated  with  the  level  of  experience  (Strater,  Endsley,  Pleban  & 
Matthews,  2001);  the  HQ  was  staffed  by  regular  officers  who  were  more  experienced 
than  the  military  students  staffing  Teams  A  and  B. 


Figure  2  -  Individual  SA  levels  by  Role  for  all  5  administrations 


The  graph  in  Figure  2  shows  each  individual’s  SA  scores  (combining  the  3  SA  levels) 
to  enable  comparisons  to  be  made  within  each  team,  as  well  as  to  check  for  extremes 
of  individual  SA  that  could  have  skewed  the  Group  SA  results  in  Figure  1.  It  was  not 
always  the  case  that  the  commander  of  a  team  would  have  higher  SA  than  his 
intelligence  and  operations  officers.  It  can  also  be  seen  that  the  intelligence  officers 
are  marginally  better  than  the  operations  officers,  which  is  consistent  with  their  roles. 


Figure  3  -  Group  SA  by  Role  for  all  5  administrations 


As  with  SAGAT,  CASA  also  allows  a  further  breakdown  of  SA  into  three  levels. 
Both  Figures  1  and  3  shows  that  average  Level  1  SA  is  higher  than  average  Level  2/3 
SA.  It  has  been  suggested  that  commanders  should  have  better  level  2/3  SA  compared 
to  level  1  as  they  are  more  focused  on  the  “big  picture”.  However,  this  was  not  the 
case  in  our  study  (see  Figure  3).  This  may  suggest  that  the  subjects  were  more  focused 
on  perceptual  elements  of  the  scenario  rather  than  integrating  elements  together  and 
projecting  into  the  future.  The  command  and  control  information  system  was  also 
designed  primarily  to  convey  factual  information  such  as  unit  strengths,  locations  and 
terrain  as  opposed  to  unit  relationships  and  status  forecasts. 


1  2  3  4  5 

Assessment 


Figure  4  -  Group  SA  by  team  and  by  administration 

Figure  4  splits  up  the  group  SA  scores  into  the  5  administrations  but  combines  the  3 
levels  of  SA.  This  gives  an  indication  of  how  SA  for  each  team  varies  across  the  5 
administrations.  The  reason  for  showing  this  graph  is  to  highlight  the  third 
administration.  Team  A  performed  relatively  poorly  in  this  assessment  compared  to 
the  other  two  teams.  Due  to  certain  exercise  events,  Team  A  completed  their  CASA 
questionnaire  approximately  one  hour  later  than  the  other  two  teams.  Because  the 
exercise  was  not  frozen  for  CASA  administrations,  events  had  changed  and  their 
answers  no  longer  corresponded  to  the  ground  truth  recorded  earlier.  This  shows  that 
the  manner  in  which  CASA  was  administered  in  the  exercise  was  prone  to  lapses  in 
timing. 


Complementary  and  Shared  SA  Levels 


Apart  from  Individual  and  Group  SA  levels,  it  is  useful  to  assess  how  well  members 
of  a  team  share  a  common  situation  picture.  On  the  other  hand,  it  can  also  be  argued 
that  good  team  performance  may  not  necessarily  call  for  all  team  members  to  possess 
the  same  situation  picture;  rather,  the  combined  awareness  of  the  team  is  what 
matters.  That  is  to  say,  as  long  as  one  member  is  aware  of  a  certain  piece  of 
information,  the  team  can  pool  their  knowledge  to  arrive  at  a  similarly  high  level  of 
understanding  and  operation. 

For  this  purpose,  measures  of  shared  and  complementary  SA  within  teams  and 
between  individuals  from  each  team  were  calculated.  In  fact,  the  existence  of  pre¬ 
defined  roles  in  a  team  (Commanders,  Intelligence  officers,  etc.)  may  support  the 
construct  of  complementary  SA  as  being  more  useful  than  shared  SA  with  the 
provision  that  knowledge  sharing  takes  place  when  necessary  according  to  military 
procedures. 


Figure  5  -  Complementary  and  Shared  SA  by  Team 


Figure  5  shows  Complementary  SA  versus  Shared  SA  (computed  as  illustrated  in 
Table  2)  within  each  3-person  team,  averaged  across  5  administrations.  It  is  evident 
that  the  two  constructs  do  not  vary  in  the  same  way  between  the  teams.  This  could  be 
a  result  of  different  team  dynamics  in  each  team.  The  results  indicate  that  Team  B’s 
shared  SA  is  half  that  of  Superior  HQ,  but  its  level  of  complementary  SA  is  nearly 
comparable.  This  may  suggest  a  lack  of  communication  between  the  team  members 
regarding  information  they  acquired  (low  Shared  SA),  however  as  a  team,  the 
different  roles  collectively  “covered  most  bases”  (high  Complementary  SA).  Further 
diagnostic  measures  such  as  tracing  communication  activity  within  and  between 
teams  would  be  required  to  determine  if  this  was  so.  Further  comparisons  against 
performance  measures  of  the  teams  can  elucidate  which  of  the  measures  of  team  SA  - 
group,  shared  or  complementary,  are  most  indicative  of  a  successful  working 
arrangement. 


As  part  of  the  effort  to  compare  distributed  versus  co-located  teams,  a  graph  was 
generated  (Figure  6)  to  look  at  shared  SA  between  counterparts  in  each  team  and 
Superior  HQ.  There  did  not  appear  to  be  a  marked  difference  between  the  Shared  SA 
for  any  of  the  roles. 
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Figure  6  -  Shared  SA  between  counterparts  in  Teams  and  HQ 
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Discussion 

The  experience  of  having  applied  CASA  in  an  actual  exercise  provided  the  research 
team  with  valuable  insights  that  enabled  refinements  to  be  made  to  several  aspects  of 
the  methodology. 

The  research  team  did  not  anticipate  that  CASA  would  be  as  manpower  intensive  as  it 
was  found  to  be  in  the  exercise.  The  use  of  a  paper  medium  required  a  team  of  people 
to  generate  questionnaires  on  word  processors,  print  multiple  copies,  and  disseminate 
them  to  the  subjects  within  a  very  short  time.  It  was  felt  that  disseminating  the 
questionnaires  was  the  most  difficult  part  of  CASA  as  the  subjects  had  to  be  located 
within  the  exercise  compound  in  a  matter  of  minutes.  For  future  field  assessments, 
CASA  can  be  modified  to  become  more  easily  self-administered,  e.g.  by  issuing  each 
subject  with  a  wireless-enabled  PDA  on  which  SA  questions  would  be  displayed  at 
the  appropriate  times,  and  the  subject  alerted  to  respond  to  the  administration.  An 
alternative  to  PDAs  would  be  to  get  subjects  to  go  to  a  room  upon  being  alerted  by  a 
P.A.  system  or  personal  beepers  to  perform  CASA  at  computer  terminals.  However, 
this  assumes  a  high  level  of  cooperation  and  adherence  to  the  protocol  on  the  part  of 
the  subjects  and  the  exercising  unit. 

The  military  SMEs  were  also  required  to  constantly  monitor  exercise  events  to  select 
suitable  administration  timings  and  craft  appropriate  SA  questions.  This  required 
them  to  be  continuously  alert  and  “on-the-job”  at  all  times.  As  this  was  the  first  time 
that  CASA  was  attempted,  there  were  instances  of  uncertainty  about  whether  an 
administration  would  be  suitable  or  if  the  exercising  unit  would  allow  an  intrusion  at 
that  time,  as  well  as  regarding  the  selection  of  questions  and  phrasing  of  new 
questions. 

Concern  was  raised  that  the  non-random  nature  of  CASA  may  render  it  susceptible  to 
bias,  since  the  selection  of  questions  and  administration  times  are  entirely  dependent 
upon  the  SMEs’  discretion.  The  questions  selected  for  each  administration  should 
attempt  to  cover  a  wide  range  of  SA  requirements  instead  of  solely  querying  about  the 
most  salient  or  prominent  situation  at  that  instance,  to  avoid  drawing  subjects’ 
attentions  to  certain  features  of  the  situation  following  the  administration.  It  is  also 
recommended  that  at  least  two  SMEs  be  employed  when  generating  questions  and 


selecting  administration  timings  so  that  the  effect  of  individual  biases  is  reduced. 
Administration  times  should  not  follow  a  predictable  pattern,  e.g.  tied  to  particular 
exercise  events  such  as  rest  periods,  so  that  subjects  cannot  anticipate  and  prepare  for 
an  administration  beforehand. 

With  regard  to  the  analysis  of  CASA  results,  although  no  statistical  tests  were  used  in 
this  case,  CASA  can  in  fact  be  subjected  to  statistical  analysis  if  certain  conditions 
were  met.  Specifically,  increasing  the  sample  size  and  number  of  administrations 
would  allow  for  statistical  testing,  although  the  amount  of  effort  required  would 
increase  accordingly.  Experience  from  this  exercise  indicates  that  CASA  is  not 
suitable  for  administration  at  too  short  intervals  due  to  the  amount  of  time  required  to 
generate  each  questionnaire.  It  remains  to  be  seen  for  future  CASA  assessments  how 
quickly  a  questionnaire  can  be  created  to  suit  rapidly  changing  scenarios. 


Conclusion  &  Future  Work 

While  CASA  has  shown  itself  to  be  a  reliable  and  effective  method  that  can  be 
applied  to  field  exercises,  it  is  acknowledged  that  data  collection  exercises  of  this 
nature  are  constrained  by  many  factors,  and  some  degree  of  trade-off  is  inevitable.  To 
obtain  strong  statistical  power  and  robustness  of  data  analysis  would  necessarily  entail 
a  correspondingly  high  level  of  intrusiveness  and  effort  on  the  part  of  the  exercise 
participants.  This  fine  balance  between  minimizing  intrusiveness  and  maximizing  data 
accuracy  will  have  to  be  sought  in  future  installations  of  CASA  as  well  as  any  other 
cognitive  assessments  performed  in  conjunction  with  it. 

To  better  validate  CASA  as  a  measure  of  SA,  other  measures  such  as  SART  (Taylor, 
1990)  for  subjective  assessment  of  SA  can  be  employed  at  the  same  time  as  CASA  to 
facilitate  comparison  between  measures.  As  mentioned  earlier,  this  was  done  to  some 
extent  in  this  exercise  and  the  findings  will  be  reported  in  a  follow-up  paper.  The  next 
logical  step  following  this  attempt  at  applying  CASA  in  an  exercise  is  to  determine 
how  various  measures  of  effectiveness  (MOE)  correlate  with  the  construct  of  SA  as 
measured  by  CASA.  For  example,  it  would  be  valuable  to  assess  if  measures  of  SA 
are  related  to  the  amount  of  time  taken  to  draw  up  a  course  of  action.  Other 
performance  measures  that  can  be  investigated  are  kill  ratios  and  quality  of  decisions 
as  rated  by  military  SMEs.  This  will  aid  in  understanding  the  significance  of  cognitive 
measures  such  as  situation  awareness  and  how  they  translate  into  and  impact 
performance  in  the  military  context. 
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Introduction 


Background 


■  In  2003,  an  exercise  was  conducted  by  the  Singapore  Armed  Forces  to 
experiment  with  distributed  but  collaborative  command  and  control 
processes 

■  Division-Level  exercise  with  human  participants  in  Brigade  and  Division- 
level  command  post;  computer  generated  forces  for  fighting  units 

■  One  of  the  experimental  conditions:  Two  command  teams  differed  in 
physical  proximity  to  HQ  but  otherwise  have  identical  communication  links 
and  information  systems 

■  Cognitive  performance  was  among  the  variables  being  investigated 


HQ 


Team  A 


Team  B 
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Introduction 


Objectives  of  study 

■  To  assess  cognitive  performance  of  command  teams  in  the 
exercise 

To  attempt  and  evaluate  various  methodologies  for  cognitive 
assessment  in  the  field 

■  To  collect  data  on  baseline  cognitive  performance  of  command 
teams 

Cognitive  performance  assessment  methods  used 

■  Situation  Awareness  (SA) 

Constructible  Assessment  For  Situation  Awareness  (CASA) 

■  Situation  Awareness  Rating  Technique  (SART) 

■  Workload 

■  NASA-TLX 

■  Communication  activity  ^ 

Video/Audio  recordings  [inPTffTl 

o  I  •••  r~Li  i 
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Constraints  of  experiments  in  field  exercises 


■  Intrusiveness  of  data  collection  to  be  minimized 

■  Exercise  events  take  precedence  over  data  collection 

■  Low  degree  of  experimental  control  -  possibility  of 
unexpected  event  injects  from  Director  of  Exercise 

■  Logistics  challenge 

Large  number  of  participants  (56) 

Long  duration  (24hrs) 

■  Physical  mobility  of  participants 
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Categories  of  SA  Measurement 


Performance  measures 

■  Global  performance  measures 

■  Subtask  performance 

■  Performance  in  response  to 
introduced  anomalies  or  events 


Direct  experimental  techniques 

■  Retrospective  measures  (e.g. 
recollection) 

■  Concurrent  measures  (e.g.  verbal 
protocols) 

■  Psycho-physiological  measures 

■  Direct  questioning  /  freeze  technique 


Subjective  measures 

■  Direct  self  rating 

■  Comparative  self 
rating 

■  Observer  rating 
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D  SO 

Endsley’s  SAGAT 


■  Direct  (explicit)  measure  of  SA  that  is  well-validated  and 
widely  applied 

■  Based  on  three  levels  of  SA  (Endsley,  1991) 

Level  1  :  Perception  of  elements  in  the  environment 
Level  2  :  Comprehension  of  the  situation 
Level  3  :  Projection  of  future  status 

■  Randomised  administration 

exercise  or  simulation  will  be  frozen 

randomly-selected  pre-determined  questions  based  on  SA 
requirements 

■  probes  into  knowledge  of  environment  and  events 

SA  is  captured  in  real-time  rather  than  post-hoc  to  reduce 
memory  errors 
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Constraints  of  experiments  in  field  exercises 


■  Intrusiveness  of  data  collection  to  be  minimized 

>  Exercise  freezes  were  not  tolerated,  administration  times  to 
be  short  (5  minutes) 

>  Irrelevant  questions  from  randomisation  are  a  concern  due  to 
limited  number  of  administrations  over  duration  of  exercise 

■  Exercise  events  take  precedence  over  data  collection 

■  Low  degree  of  experimental  control  -  possibility  of 
unexpected  event  injects  from  Director  of  Exercise 

■  Logistics  challenge 

Large  number  of  participants  (56) 

>  Nine  key  participants  identified  for  objective  SA  assessment 
Long  duration  (24hrs) 

Physical  mobility  of  participants 

>  Paper-based  administration 
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Workflow  of  CASA 


Observation  of  Exercise 


Progressive  buildup  of 
database  of  CASA 
question  templates 


Selection  of 

Selection/generation  of 

administration  time 

relevant  questions 

" 


Generation  of 
questionnaires 


Obtain  answers  from 

Dissemination  to 

ground  truth 

i 

subjects 

Data  collection  and  Analysis 
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Examples  of  CASA  questions 


Question  Formats 

Level  1  SA 

1.  Mark  the  location  of  [red/blue]  unit  on 
map. 

2.  What  is  the  current  size  force  of 
[red/bluej  unit?  (e.g.  “Coy+”) 

Level  2  SA 

1.  What  is  the  most  critical  additional  asset 
that  [blue]  unit  requires  to  carry  out  its 
mission? 

2.  Which  hostile  unit  currently  poses 
highest  threat  priority  to  this  [blue]  unit? 

Level  3  SA 

1.  When  is  the  earliest  projected  time  for 
the  securing  of  [location]? 

2.  Is  [red]  unit  likely  to  be  in  contact  with 
[blue]  unit  by  [time]  ? 


Answer  Formats 

•  Only  1  correct  answer  per  question 

•  Multiple  choice  questions 

•  Map-based 

•  Open-ended  (constrained  by  context) 


9th  ICCRT Symposium 


Copyright  ©  2004  by  DSO  National  Laboratories.  All  Rights  Reserved. 


DSO 


CASA  Analysis 


Individual  X  SA  =  3  4-  5  =  60% 

Group  XY  SA  =  7  -h  1 0  =  70% 

Group  XYZ  SA  =  9  h-  15  =  60% 

Shared*  XY  SA  =  4  +  5  =  80% 

Shared*  YZ  SA  =  2  +  5  =  40% 

Shared*  XYZ  SA  =  1  +  5  =  20% 
Complementary  SA  of  XYZ  =  4  +  5  =  80% 

*  In  this  illustration,  wrong  answers  contribute  to  Shared  SA  if  respondents  answered  similarly 


Question 

Xs 

answer 

Y’s 

answer 

Z’s 

answer 

i 

(aK 

(a)Z 

(0* 

2 

(b)x 

(b)* 

(c)x 

3 

(a)Z 

(aK 

(a)  Y 

4 

(c)x 

(d)Z 

(d)Y 

5 

(dK 

(d)^ 

MJL 

Shared  SA  =  Intersection 
Complementary  SA  =  Union 


Legend 

1- (a)  represents  Question  1 
being  answered  correctly  with 
option  (a) 

2- (b)  represents  Question  2 
being  answered  incorrectly 
with  option  (b) 
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Applying  CASA 


Pre-exercise  preparation 

■  Nine  subjects  identified 

(Commander,  Operations,  Intelligence)  x  3  teams 

■  Generation  of  CASA  question  templates 

Interviews  with  2  military  Subject  Matter  Experts  (SMEs) 

Identified  information  requirements  of  Commanders, 
Operations  and  Intelligence  officers  to  complete  their  tasks 

Categorised  requirements  into  three  levels  of  SA 
■  Translated  information  requirements  into  questions 
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Applying  CASA 


During  the  exercise 

■  Questionnaires  were  constructed  1-2  hours  from  the  time  it 
was  decided  to  have  an  administration;  jointly  by  DSO 
researchers  and  SMEs 

■  Disseminated  to  subjects  within  5  minutes  of  time  of 
administration 

■  Subjects  completed  questionnaires  (10  or  less  questions) 
within  5  minutes 

■  Answers  to  queries  (ground  truth)  were  recorded  at  the 
appropriate  times 

■  Questionnaires  were  graded  against  the  answers 
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DSO 


Breakdown  of  CASA  questions  administered 


Projected  location  of 
[blue]  unit  (map) 
2% 

Projected  time  for 
[event]  to  happen 
9% 


Erroneous  questions 
4% 


Current  mission  /  status 
of  [red]  unit 
2% 


Impact  of  [blue] 
mission  on  [blue]  unit 
9% 


Others  (Lvl  2) 
2% 


Relationship  between 
[red/civ]  unit  to  [blue] 
unit 
11% 

Location  of  [red]  units 
(map) 

8% 


□ 

Level  1  SA 

□ 

Level  2  SA 

□ 

Level  3  SA 

■ 

Decision  Making 
type  questions 

Current  mision  /  status 
of  [blue]  unit 
9% 
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Current  Size  Force  of 
[red]  unit 
4% 
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Others  (lvl  1) 
4% 


DSO 


Results 
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CMD  IN  T  CPS  CMD  I  NT  OFS 


SA  level  of  each  individual,  averaged  across  5  assessments 
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DSO 


Results 
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HQ  Team  A  Team  B 


Group  SA  by  Team 


Level  1  SA  >  Level  2/3  SA 


Possible  explanation: 
Level  1  SA  information 
elements  are  currently 
better  represented  or 
conveyed  compared  to 
Levels  2/3  SA 
information  elements. 

Need  to  enhance  Level 
2/3  SA  representation 
and  conveyance  (e.g 
visualization,  symbology, 
decision  support,  etc) 
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DSO 


Results 


□  Level  1  SA 

a  Lev  el  2/3  SA 

60  <- 
ED  -■ 

4D  -■ 

3D  -■ 

2D  -■ 

ID  -■ 

D  -■ 

Conrnander  Intelligence  Operations 


Role 


Group  SA  by  Role 


Result  consistent  with  the 
roles  of  intelligence  and 
operations  officers 

It  has  been  suggested  that 
Commanders’  Level  2/3 
SA  >  Level  1  SA  as  they 
focus  on  the  big  picture. 
However  this  is  not 
supported  by  the  results 


1 .  Intelligence  SA  >  Operations  SA 

2.  Level  1  SA  >  Level  2/3  SA 
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DSO 

Results 


SA  level  of  Team  A  >  Team  B,  except  for  Administration  3. 
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DSO 


Results 


100% 

90% 

80% 

70% 
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50% 

40% 

30% 

20% 

10% 

0% 


□  Complementary  SA 
■  Shared  SA 


HQ  Team  A  Team  B 


Shared  SA  measures  how  much  information 
each  member  has  in  common  with  others 


Suggests  that 
information  was  not 
shared  as  freely  as 
may  be  desirable 

Need  to  enhance  both 
(co-located)  physical 
communication  and 
(distributed)  electronic 
tools  to  support 
communication 


Complementary  SA  is  a  measure  of  team 
SA  assuming  all  members  readily  share 
information  -  may  be  a  more  appropriate 
measure  in  the  military  C2  context  with 
assigned  roles 
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Issues 


■  Possible  introduction  of  bias  on  the  part  of  SMEs 

With  regards  to: 

(i)  Selection  of  questions 

(ii)  Selection  of  administration  times 

>  Mitigated  with  multiple  SMEs  or  third-party  researchers 

■  Labour-intensive  method 

■  Constant  monitoring  of  events 

■  Generating  and  administering  questionnaires  in  a  short  time  frame 

>  Electronic  means  (e.g.  wireless  devices)  to  replace  paper 
administration 

■  Diagnostic  capability  -  allows  experimenters  to  select  or  generate 
SA  questions  to  probe  participants  on  specific  issues 


9th  ICCRT Symposium 


Copyright  ©  2004  by  DSO  National  Laboratories.  All  Rights  Reserved. 


Reliability  and  Validity  of  CASA 


■  Reliability  of  CASA 

■  Careful  phrasing  of  question  templates 

Generation  of  actual  questions  from  the  question  templates 

■  Validity  of  CASA 

■  CASA  results  (objective  measure)  were  compared  with  SART  results 
(subjective  measure)  but  they  did  not  correspond 

Possible  Reason  -  Overestimation  of  own-self’s  performance 

‘Slices’  of  Objective  Measure  of  SA  Subjective  Measure  of  SA 


Possible  Reason  -  Objective  measures  of  SA  measures  a  slice  of  an 
individual’s  SA  over  time,  whereas  subjective  measures  of  SA  may  be 
more  inclusive  base  on  an  individual’s  overall  experience. 
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Future  Work 


■  Develop  new  measures  of  SA  comprising  both  objective 
and  subjective  components 

■  Correlate  SA  with  measures  of  effectiveness  (MOE)  e.g. 
time  taken  to  formulate  plans  or  quality  of  decisions  made 

■  Correlate  SA  with  workload  in  command  teams 

■  Further  validation  of  CASA  in  other  experiments  and 
settings 
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Questions? 
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